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Artificial Intelligence: Opportunities and Risks 


Executive Summary 


Artificial intelligence (Al) and increasingly complex algorithms currently influence our lives and our civilization more than 
ever before. The areas of Al application are diverse and the possibilities far-reaching, and thanks to recent improvements 
in computer hardware, certain Al algorithms already surpass the capacities of today’s human experts. As Al capacity im- 
proves, its field of application will continue to grow. In concrete terms, it is likely that the relevant algorithms will start opti- 
mizing themselves to an ever greater degree and may one day attain superhuman levels of intelligence. This technological 
progress is likely to present us with historically unprecedented ethical challenges. Many experts believe that, alongside 
global opportunities, Al poses global risks surpassing those of e.g. nuclear technology (whose risks were severely under- 
estimated prior to their development). Furthermore, scientific risk analyses suggest that high potential damages resulting 
from Al should be taken very seriously—even if the probability of their occurrence were low. 


Current 


In narrow, well-tested areas of application, such as driverless cars and certain areas of medical diagnostics, the superiority 
of Als over humans is already established. An increased use of technology in these areas offers great potential, including 
fewer road traffic accidents, fewer mistakes in the medical treatment and diagnosing of patients, and the discovery of 
many new therapies and pharmaceuticals. In complex systems where several algorithms interact at high speed (such as in 
the financial market or in foreseeable military uses), there is a heightened risk that new Al technologies will be misused, or 
will experience unexpected systematic failures. There is also the threat of an arms race in which the safety of technological 
developments is sacrificed in favor of rapid progress. In any case, it is crucial to know which goals or ethical values ought 
to be programmed into Al algorithms and to have a technical guarantee that the goals remain stable and resistant to 
manipulation. With driverless cars, for instance, there is the well-known question of how the algorithm should act if a 
collision with several pedestrians can only be avoided by endangering the passenger(s), not to mention how it can be 
ensured that the algorithms of driverless cars are not at risk of hacking systematic failure. 


Measure 1 The promotion of a factual, rational discourse is essential so that cultural prejudices can be dismantled 
and the most pressing questions of safety can be focused upon. 


Measure 2 Legal frameworks must be adapted so as to include the risks and potential of new technologies. Al manu- 
facturers should be required to invest more in the safety and reliability of technologies, and principles like predictabil- 
ity, transparency, and non-manipulability should be enforced, so that the risk of (and potential damage from) unex- 
pected catastrophes can be minimized. 


Mid-term 


Progress in Al research makes it possible to replace increasing amounts of human jobs with machines. Many economists 
assume that this increasing automation could lead to a massive increase in unemployment within even the next 10-20 
years. It should be noted that while similar predictions in the past have proved inaccurate, the developments discussed 
here are of a new kind, and it would be irresponsible to ignore the possibility that these predictions come true at some 
point. Through progressive automation, the global statistical average living standard will rise; however, there is no guar- 
antee that all people—or even a majority of people—will benefit from this. 
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Measure 3 Can we as a society deal with the consequences of Al automation in a sensible way? Are our current social 
systems sufficiently prepared for a future wherein the human workforce increasingly gives way to machines? These 
questions must be clarified in detail. If need be, proactive measures should be taken to cushion negative developments 
or to render them more positive. Proposals like an unconditional basic income or a negative income tax are worth 
examining as possible ways to ensure a fair distribution of the profits from increased productivity. 


Long-term 


Many Al experts consider it plausible that this century will witness the creation of Als whose intelligence surpasses that 
of humans in all respects. The goals of such Als could in principle take on any possible form (of which human ethical 
goals represent only a tiny proportion) and would influence the future of our planet decisively in ways that could pose 
an existential risk to humanity. Our species only dominates Earth (and, for better or worse, all other species inhabiting 
it) because it currently has the highest level of intelligence. But it is plausible that by the end of the century, Als will be 
developed whose intelligence compares to ours as ours currently compares to, say, chimpanzees. Moreover, the possi- 
bility cannot be excluded that Als also develop phenomenal states—i.e. (self-)consciousness, and in particular subjective 
preferences and the capacity for suffering—in the future,which would confront us with new kinds of ethical challenges. In 
view of the immediate relevance of the problem and its longer-term implications, considerations of Al safety are currently 
highly underrepresented in politics as well as research. 


Measure 4 Itis worth developing institutional measures to promote safety, for example by granting research funding 
to projects which concentrate on the analysis and prevention of risks in Al development. Politicians must, in general, 
allocate more resources towards the ethical development of future-shaping technologies. 


Measure 5 Efforts towards international research collaboration (analogous to CERN’s role in particle physics) are to 
be encouraged. International coordination is particularly essential in the field of Al because it also minimizes the risk 
of a technological arms race. A ban on all risky Al research would not be practicable, as it would lead to a rapid and 
dangerous relocation of research to countries with lower safety standards. 


Measure 6 Certain Al systems are likely to have the capacity to suffer, particularly neuromorphic ones as they are 
structured analogously to the human brain. Research projects that develop or test such Als should be placed under 
the supervision of ethical commissions (analogous to animal research commissions). 


Introduction 


The pursuit of knowledge runs asa governing principle 
through human history. Whenever societies have under- 
gone significant changes in their dynamics and structure, 
this has normally been the result of new technological 
inventions. Around two million years separate the first 
use of stone tools from the historic moment when Homo 
sapiens invented art and began to paint images on cave 
walls. Another thirty thousand years passed before the 
rise of arable farming and permanent settlement. The first 
symbols appeared a few thousand years after that, fol- 
lowed closely by the first written scripts. Then, around 
four hundred years ago, development began speeding up. 
The microscope was invented in the seventeenth century; 
industrialization in the nineteenth century enabled the 
first cities of a million people; and during the last cen- 
tury alone, the atom was split, humans set foot on the 
Moon, and the computer was invented. Since then, the 
processing capabilities and energy efficiency of computers 
have doubled at regular intervals [1]. But while technolog- 
ical progress often develops exponentially, the same is not 
true for human intellectual abilities. 

In recent years, countless renowned scientists and en- 
trepreneurs have warned of the urgent significance of Al, 
and how important it is that policy makers tackle the chal- 
lenges raised by Al research [2]. Exponents of this move- 
ment for Al safety include Stuart Russell [3], Nick Bostrom 
[4], Stephen Hawking [5], Sam Harris [6], Max Tegmark [7], 
Elon Musk [8], Jann Tallinn [9] and Bill Gates [10]. 

In certain domain-specific areas, Als have already 
reached or even overtaken human levels on several oc- 


Advantages and risks of current Als 


Our individual lives and our civilization as a whole are 
governed to an ever-increasing extent by algorithms and 
Well- 
known examples include such ubiquitous things as smart- 


domain-specific artificial intelligence (Als) [19]. 


phones, air traffic control systems [20] and internet search 
engines [21]. Financial markets, too, are dependent on al- 
gorithms which are too large and complex for any single 
human being to fully understand [22, 23]. The operation 
of such algorithms, for the most part, proceed without in- 
cident, but there is always the possibility that an unlikely 
“black swan” event [24] might occur, threaten to plunge 
the whole system into chaos. We have already witnessed 
one such event: in 2010, an unexpected "flash crash" in 
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casions. In 1997 the computer Deep Blue beat the reign- 
ing world champion Garry Kasparov at chess [11]; in 2011 
Watson beat the two best human players on the language- 
based game show Jeopardy! [12]; and in 2015 the first vari- 
ant of poker, Fixed Limit Holdem heads-up, was game the- 
oretically fully solved by Cepheus [13]. Meanwhile, artifi- 
cial neural networks can compete with human experts in 
the diagnosis of cancer cells [14] and are also more or less 
approaching human levels in the recognition of handwrit- 
ten Chinese characters [15]. Back in 1994, a self-learning 
backgammon program reached the level of the world’s 
best players by finding strategies that had never before 
been played by humans [16]. By now, there even exist algo- 
rithms that can independently learn many different games 
from scratch and thereby reach (or surpass) human levels 
[17, 18]. With these developments, we are slowly getting 
closer to a general intelligence, which at least in principle 
can solve problems of all sorts independently. 

With great power comes great responsibility. Technol- 
ogy is in itself just a tool; what matters is how we use it. 
The use of existing Als is already presenting us with consid- 
erable ethical challenges, which will be illuminated in the 
next section of this paper. The following chapter will out- 
line developments in economic automation, and explain 
the mid-term prognosis that Al research will give rise to a 
significant restructuring of the labor market. Finally, the 
two last chapters will discuss the long-term and existen- 
tial risks of Al research in relation to the possible creation 
of (super)human intelligence and artificial consciousness. 


a US stock market left the financial world dumbfounded. 
The crash occurred as a result of computer algorithms in- 
teracting with the financial market in an unforeseen man- 
ner [25, 26]. Within minutes, important shares lost more 
than 90% of their worth and then quickly returned to their 
high initial value. If such an event were to take place in 
a military context, a comparable “return to initial condi- 
tions” would be improbable [27]. To prevent devastating 
failures of this sort, it seems generally advisable to invest 
considerably more resources into the safety and reliability 
of Als. Unfortunately, current economic incentives seem 
to favor increased Al capacity far more than safety. 
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Four criteria for the construction of Als 


Safety is essential to the construction of any sort of ma- 
chine. However, new ethical challenges arise when con- 
structing domain-specific Als capable of taking over cog- 
nitive work in social dimensions—work that, until now, 
has been carried out by humans. For instance, an al- 
gorithm that judges the credit rating of bank customers 
might make decisions that discriminate against certain 
groups in the population (without this being explicitly pro- 
grammed). Even technologies that simply replace exist- 
ing actions could introduce interesting challenges for ma- 
chine ethics [28]: driverless cars, for instance, raise the 
question of which criteria should be decisive in the case of 
an imminent accident. Should the vehicle ensure the sur- 
vival of the passengers above all else or should it, in the 
case of an unavoidable accident, prioritize keeping the to- 
tal number of casualties as low as possible [29]? 

Because of this, both Al theorist Eliezer Yudkowsky and 
philosopher Nick Bostrom have suggested four principles 
which should guide the construction of new Als [30]: 1) the 
functioning of an Al should be comprehensible and 2) its 
actions should be basically predictable. Both of these cri- 
teria must be met within a time frame that enables the re- 
sponsible experts to react in time and veto control in case 
of a possible failure. In addition, 3) Als should be impervi- 
ous to manipulation, and in case an accident still occurs, 4) 
the responsibilities should be clearly determined. 


Advantages of (domain specific) artificial intelligence 


In principle, algorithms and domain-specific Als bring 
many advantages. They have influenced our lives for 
the better and are expected to keep doing so at an ever- 
increasing rate in the future, provided that the necessary 
precautions are taken. Here we will discuss two instructive 
examples. 

Driverless cars are no longer science fiction [31, 32]; 
they’ll be commercially available in the foreseeable future. 
The Google Driverless Car, which is driven completely by 
autonomous Al algorithms, took its first test drive in the 
USA back in 2011 [33, 34]. Besides the time gained for work 
or relaxation, a second advantage to driverless cars con- 
sists in their higher safety. In 2010, 1.24 million people died 
worldwide in traffic accidents, nearly exclusively because 
of human error [35]. Countless human lives could there- 
fore be saved every year, because driverless cars are al- 
ready significantly safer than vehicles driven by humans 
[36, 37]. 

Naturally, a large number of people remain skeptical 


regarding driverless cars, mainly because they underesti- 
mate the safety benefits thereof whilst at the same time 
overestimating their own driving abilities. As an illustra- 
tion of this latter point, one study came to the conclusion 
that 93% of all American drivers believe that their driving 
abilities are above the median [38]—which is statistically 
impossible. Unrealistic optimism [39] and the illusion of 
control [40] possibly also bias people towards underes- 
timating the risks when they themselves are behind the 
wheel [41,42]. 

Doctors, too, overestimate their abilities [43], which in 
the worst case can lead to deadly mishaps. In the USA 
alone, between an estimated 44,000 and 98,000 people 
die each year in hospitals because of treatment mistakes 
[44]. In this context, IBM’s Watson [45] is a welcome devel- 
opment. This Al gained fame in 2011 when it beat the best 
human players on the quiz show Jeopardy! [12]. Watson 
isn’t just better than humans in quiz shows, however. Hos- 
pitals have been able to hire Watson’s computing power 
since 2014 for cancer diagnosis and other complex pattern- 
recognition tasks. Because “Doctor Watson” can rapidly 
collect and combine enormous quantities of information, 
it has partially overtaken the diagnostic skills of its human 
colleagues [46, 47]. 

The fact that a current Al can make more accurate med- 
ical diagnoses than human doctors may seem surprising at 
first, but it has long been recognized that statistical infer- 
ences are superior to clinical judgments by human experts 
in most cases [48, 49]. Seeing as Als like Watson are ideal 
for making statistical inferences, it follows that using com- 
puters for certain types of diagnosis can save lives. 


Cognitive biases: to err is human 


One reason why human experts are less competent than 
Als at statistical inferences is the aforementioned (and, un- 
fortunately, all too human) tendency to overestimate one’s 
own abilities. This tendency is known as overconfidence 
bias [50] and is just one of many documented cognitive bi- 
ases that can lead to systematic errors in human thinking 
[51, 52]. Als, on the other hand, can be built so as to avoid 
cognitive biases altogether. In principle, increasing confi- 
dence in the predictions of Als could lead to a significantly 
more rational and efficient approach to many social and 
political challenges, provided they are made safely and ac- 
cording to comprehensible criteria. The problem here lies 
in using the strengths of Al without at the same time giving 
up human autonomy in the corresponding systems. 


Conclusion and outlook 


Irrational fears towards new and basically advantageous 
technologies are widespread, both now and in the past 
[53]. Such “technophobia” may also be one of the rea- 
sons that Watson or driverless cars are met with skepti- 
cism. However, being wary of kinds of technology is not al- 
ways irrational. Most technologies can be used to the ben- 
efit of humanity, but can also be dangerous when they fall 
into the wrong hands, or when insufficient care is taken for 
safety and unforeseen side effects. 
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This also holds for artificial intelligence: driverless 
cars could make our lives easier and save human lives, 
but complex computer algorithms can also cause the 
stock market to crash unexpectedly. While the risks from 
domain-specific Als appear limited in the near future, 
there are long-term developments to take into consider- 
ation: in the not-so-distant future, artificial intelligence 
could in principle pose an existential threat, similar in 
scope to the pandemic risks associated with biotechnol- 
ogy [54, 55, 4]. 


Recommendation 1 — Responsible approach: As with all other technologies, care should be taken to ensure that the 


(potential) advantages of Al research clearly outweigh the (potential) disadvantages. The promotion of a factual, ratio- 


nal discourse is essential so that irrational prejudices and fears can be broken down. Current legal frameworks have 


to be updated so as to accommodate the challenges posed by new technologies. The four principles described above 


should be followed for every extensive use of Als [30]. 


Automation and unemployment 


In light of recent successes in the field of machine learning 
and robotics, it seems there is only a matter of time until 
even complicated jobs requiring high intelligence could be 
comprehensively taken over by machines [56]. 

If machines become quicker, more reliable and cheaper 
than human workers in many areas of work, this would 
likely cause the labour market to be uprooted on a scale 
not seen since the Industrial Revolution. According to 
economists like Cowen [57], McAfee and Brynjolfsson [58], 
technological progress will widen the income gap even fur- 
ther and may lead to falling incomes and rising unemploy- 
ment in large segments of the population. 

A 2013 analysis concluded that it will likely be possible 
to automate 47% of all jobs in the USA within 10-20 years 
[59]. The hardest jobs to automate are those which require 
high levels of social intelligence (e.g. PR consultation), cre- 
ativity (e.g. fashion design) and/or sensitive and flexible 
object manipulation (e.g. surgery). In these domains, the 
state of Al research is still far below the level of human ex- 
perts. 


Advantages and disadvantages to automation by com- 
puters 


Those who will benefit the most from technological 
progress are the people and nations that understand how 
to make use of new technological opportunities and the 
corresponding flood of “big data” [60]. In particular, coun- 
tries with well-trained computer specialists are expected 
to prosper in the face of technological progress. More- 


over, it is likely that a thorough understanding of the ways 
in which various computer algorithms compare to hu- 
man decision-making and working abilities—as well as the 
(dis)advantages of each—will become increasingly impor- 
tant in the future, thus necessitating high standards of ed- 
ucation [61]. 

Following the automation of the production and ser- 
vice industries, one might expect only the entertainment 
industry to remain; yet here, too, we are already witness- 
ing extensive changes. With flawless computer graphics, 
novel entertainment technologies, and countless smart- 
phone apps all becoming increasingly affordable, the ad- 
dictive pull of videogames and internet usage is rising [62]. 
While we have not yet been able to research the long-term 
social and psychological consequences of this develop- 
ment, several factors currently indicate that these trends 
are profoundly changing our social behavior [63], atten- 
tion spans, and childhood development [64]. These ef- 
fects may be amplified by the increasing use of virtual re- 
ality technology, which is already available to consumers. 
As these become increasingly detailed and realistic, they 
may blur the user’s boundaries between reality and simu- 
lation, thereby invading deeper into our everyday experi- 
ence. The consequences of more regular immersion in vir- 
tual realities—including experiences like body-transfer il- 
lusions, in which subjective awareness is temporarily pro- 
jected into a virtual avatar [65]—should receive greater at- 
tention. 

While the entertainment industry does offer significant 
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opportunities for better education through personalized 
Al teaching and the gamification of learning material [66], 
it also increases the risk that a growing proportion of 
youngpeoplewillhavetrouble completingtheireducation 
due to a pathological addiction to video games and/or the 
internet [67]. 


Utopias and dystopias 


Technological progress increases societal productivity 
[68], in turn raising the average standard of living [69]. If 
more work is carried out by machines, this frees up time for 
leisure and self-development for humans—at least those 
in a position to profit from it. However, a drawback to in- 
creasing automation could be that the increases in pro- 
ductivity go along with increasing social inequality so that 
arise in the mean standard of living doesn’t coincide with 
arise in the median quality of life. Experts like the MIT eco- 
nomics professor Erik Brynjolfsson even worry that tech- 
nological progress threatens to make the lives of a major- 
ity of people worse [70]. 

Ina competitive economy where Al technology has pro- 
gressed to the point where many jobs are done by ma- 
chines, the income for automatable human work will fall 
[58]. Without regulation, the incomes of many people 
could sink below subsistence level. Social inequality may 
rise sharply if economic output were to increase more 
rapidly than the wages needed to effect redistribution. 
To counteract this development, McAfee and Brynjolfsson 
suggest that limiting certain jobs to humans should be 
subsidized. Additional options for ensuring fair distribu- 


tion of advantages from technological progress amongst 
the whole population include unconditional basic income, 
and a negative income tax [71, 72] 

Some experts also warn of future scenarios in which 
the projected changes are even more drastic. For ex- 
ample, the economist Robin Hanson expects that it 
will be possible within this century to digitally run hu- 
man brain simulations—so-called whole brain emulations 
(WBEs) [73]—in virtual reality. WBEs would be repro- 
ducible, and could (assuming that sufficient hardware is 
available) run many times faster than a biological brain, 
consequently implying a huge increase in labor efficiency 
[74]. Hanson predicts that in such a case, there would 
be a “population explosion” amongst WBEs, who could 
be used as enormously cost-efficient workers [75]. Han- 
son’s speculations are contested [61], and it should not be 
assumed that they sketch out the most likely future sce- 
nario. Current research in this field, such as the Blue Brain 
Project at ETH Lausanne, is still very far from the first brain 
simulations—never mind supplying them in real time (or 
even faster) with inputs from a virtual reality. However, it is 
important to keep hardware developments in mind in rela- 
tion to the possibility of WBEs. If the scenario sketched out 
by Hanson were to occur, this would be of great ethical rel- 
evance. For one thing, many humans replaced by complex 
simulations could become unemployed; for another, there 
is the question whether the WBEs deployed would have 
phenomenal consciousness and subjective preferences— 
in other words, whether they would experience suffering 
as a result of their (potentially forced) labor. 


Recommendation 2 — Forward thinking: As in the case of climate change, incentives should be set for researchers 
and decision makers to deal with the consequences of Al research; only then can the foundations of precautionary 
measures be laid. In particular, specialist conferences should be held on Al safety and on assessing the consequences 
of Al, expert commissions should be formed, and research projects funded. m 


Recommendation 3 — Education: The subsidization of human work, an unconditional basic income, and a negative 
income tax have all been proposed as measures to cushion the negative social impacts of increased automation. Re- 
search should be conducted toward finding additional options, as well as identifying which set of measures has the 
maximum effect. Moreover, advantages and disadvantages must be systematically analyzed and discussed at a politi- 
cal level, and research grants should be established in order to answer any empirical questions that will inevitably arise 
as a result of this discussion. a 


Recommendation 4 — Transparency over new measures: The subsidisation of human work, an unconditional basic 
income or a negative income tax have been proposed as measures to cushion the negative social impacts of increasing 
automation. It is worth clarifying which further options exist and which set of measures has the maximum effect. In 
addition, advantages and disadvantages must be systematically analysed and discussed at a political level. Research 
grants should be established to answer the empirical questions thrown up by this discussion. = 
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General intelligence and superintelligence 


General intelligence measures an agent'sability to achieve 
goals in a wide range of environments [76, 77]. This kind 
of intelligence can pose a (catastrophic) risk ifthe goals of 
the agent do not align with our own. If a general intelli- 
gence reaches a superhuman level, it becomes a superin- 
telligence; that is, an algorithm superior to human intel- 
ligence in every way, including scientific creativity, “com- 
mon sense”, and social competence. Note that this defini- 
tion leaves open the question of whether or not a superin- 
telligence would have consciousness [78, 79]. 


Comparative advantages of general artificial intelligence 
over humans 


Humans are intelligent, two-legged “bio-robots” possess- 
ing a conscious self-awareness, and were developed over 
billions of years of evolution. These facts have been used 
argue that the creation of artificial intelligence may not be 
so difficult, [80, 81, 82] seeing as Al research can be con- 
ducted in a faster, more goal-oriented way than evolution 
(which only progresses through the slow accumulation of 
successive generations). Alongside the fact that evolution 
is a precondition for the feasibility of Als, it naturally also 
permits directed human research to borrow from biologi- 
cal design and thereby proceed considerably faster. 

Compared to the biological brain of a person, computer 
hardware offers several advantages[4, p. 60]: the basic 
computational elements (modern microprocessors) “fire” 
millions of times faster than neurons; signals are trans- 
mitted millions of times faster; and a computer can store 
considerably more basic computational elements in total 
(a single supercomputer can easily take up an entire fac- 
tory floor). A future digital intelligence would also have big 
advantages over the human brain in relation to software 
components [4, pp. 60-61]: for instance, it is easy to both 
modify and multiply, meaning that potentially relevant in- 
formation can be called upon at any time. In a few impor- 
tant areas such as energy efficiency, resilience to purely 
physical damage, and graceful degradation [83], artificial 
hardware still lags behind the human brain. In particular, 
there is still no direct relation between thermodynamic ef- 
ficiency and complexity reduction at the level of informa- 
tion processing [84, 85], but this may change as computer 
hardware improves in coming decades. 

In view of these comparative advantages and the pre- 
dicted rapid improvement of hardware [86] and software, 
it seems probable that human intelligence will someday 
be overtaken by that of machines. It is important to assess 


more precisely how and when this could take place, and 
where the implications of such a scenario lie. 


Timeframes 


Different experts in the area of Al have considered the 
question of when the first machines will reach the level of 
human intelligence. Asurvey of the hundred most success- 
ful Al experts, measured according to a citation index, re- 
vealed that a majority consider it likely that human-level 
Al will be developed within the first half of this century [4, 
p. 19]. The belief that humans will create a superintelli- 
gence by the end of this century, as long as technologi- 
cal progress experiences no large setbacks (as a result of 
global catastrophes), was also held by the majority of ex- 
perts [4, p. 20]. The variance among these estimates is 
high: some experts are confident that there will be ma- 
chines with at least human levels of intelligence no later 
than 2040; (fewer) other experts think that this level will 
never be reached. Even if one makes a somewhat conser- 
vative assumption, accounting for the tendency of human 
experts to be overconfident in their estimates [87, 88], it 
would still be inappropriate to describe superintelligence 
as mere “science fiction” in the light of such widespread 
confidence among relevant experts. 


Goals of a general intelligence 


Asa rational agent, an artificial intelligence strives towards 
just what its goals/goal function describes [89]. Whether 
an artificial intelligence will act ethically, that is, whether 
it will have goals which are not in conflict with the interests 
of humans and other sentient beings, is completely open: 
an artificial intelligence can in principle follow all possi- 
ble goals [90]. It would be a mistaken anthropomorphi- 
sation to think that every kind of superintelligence would 
be interested in ethical questions like (typical) humans. 
When we build an artificial intelligence, we also establish 
its goals, explicitly or implicitly. 

These claims are sometimes criticized on the grounds 
that any attempt to direct the goal of an artificial intelli- 
gence according to human values would amount to “en- 
slavement,” because our values would be forced upon the 
Al [91]. However, this criticism rests on a misunderstand- 
ing, as the expression “forced” suggests that a particular, 
“true” goal already exists, one the Al has before it is cre- 
ated. This idea is logically absurd, because there is no 
pre-existing agent “receiving” the goal function in the first 
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place, and thus no goal independent of the processes that 
have created an agent. The process that creates an intel- 
ligence determines inevitably its functioning and goals. /f 
we intend to build a superintelligence, then we, and noth- 
ing and nobody else, are responsible for its goals. Fur- 
thermore, it is also not the case that an Al must experi- 
ence any kind of harm through the goals that we inevitably 
give it. The possibility of being harmed in an ethically rele- 
vant sense requires consciousness, which we must ensure 
is not achieved by a superintelligence. Parents inevitably 
form the values and goals of their children’s “biological in- 
telligence” in a very similar way, yet this does obviously 
not imply that children are thereby “enslaved” in an un- 
ethical manner. Quite the opposite: we have the greatest 
ethical duty to impart fundamental ethical values to our 
children. The same is true for the Als that we create. 

The computer science professor Stuart Russell warns 
that the programming of ethical goals poses a great chal- 
lenge [3], both on a technical level (how would complex 
goals in a programming language be written so that no un- 
foreseen consequences resulted?) and on an ethical level 
(which goals anyhow?). The first problem is called the 
value-loading problem in the literature [92]. 

Although the scope of possible goals of a superintel- 
ligence is huge, we can make some reliable statements 
about the actions they would take. There is a range of in- 
strumentally rational subgoals that are useful for agents 
with highly varied terminal goals. These include goal- and 
self-preservation, increasing one’s intelligence, and re- 
source accumulation [93]. If the goal of an Al were altered, 
this could be as negative (or even more so) to the achieve- 
ment of its original goal as the destruction of the Al itself. 
Increased intelligence is essentially just an ability to reach 
goals in a wider range of environments, and this opens 
up the possibility of a so-called intelligence explosion, in 
which an Al rapidly undergoes an enormous increase in its 
intelligence through recursive self-improvement [94, 95] 
(a concept first described by I.J. Good [96] which has since 
been formalized in concrete algorithms [97].) Resource ac- 
cumulation and the discovery of new technologies give the 
Al more power, which in turn serves better goal achieve- 
ment. If the goal function of a newly developed superintel- 
ligence ascribed no value to the welfare of sentient beings, 
it would cause reckless death and suffering wherever this 
was useful for its (interim) goal achievement. 

One could tend towards the assumption that a super- 
intelligence poses no danger because it is only a com- 
puter, which one could literally unplug. By definition, how- 
ever, asuperintelligence would not be stupid; if there were 


any probability that it would be unplugged, a superintelli- 
gence could initially behave itself as the makers wished it 
to, until it had found out how to minimize the risk of an in- 
voluntary shutdown [4, p. 117]. It could also be possible for 
a superintelligence to circumvent the security systems of 
big banks and nuclear weapon arsenals using hitherto un- 
known gaps in security (so-called zero day exploits), and in 
this way to blackmail the global population and force it to 
cooperate. As mentioned earlier, in such a scenario a “re- 
turn to the initial situation” would be highly improbable. 


What is at stake 


In the best-case scenario, a superintelligence could solve 
countless problems for humanity, helping us overcome 
the greatest scientific, ethical, ecological and economic 
challenges of the future. If, however, the goals of a super- 
intelligence were incompatible with the preferences of hu- 
man beings or any other sentient beings, it would amount 
to an unprecedented existential threat, potentially caus- 
ing more suffering than any preceding event in the known 
universe [98]. 


Rational risk management 


In decision situations where the stakes are very high, the 
following principles are of crucial importance: 


1. Expensive precautions can be worth the cost even 
for low-probability risks, provided there is enough to 
win/lose thereby [89]. 


2. When there is little consensus in an area amongst ex- 
perts, epistemic modesty is advisable. That is, one 
should not have too much confidence in the accu- 
racy of one’s own opinion either way. 


The risks of Al research are of a global nature. If Al re- 
searchers fail to transfer ethical goals to a superintelli- 
gence in the first attempt, there quite possibly won’t be 
a second chance. It is absolutely tenable to estimate the 
long-term risks of Al research as even greater than those 
of climate change. In comparison to climate change, how- 
ever, Al research is receiving very little attention. With this 
paper, we want to emphasize that it is therefore even more 
valuable to invest considerable resources into Al safety re- 
search. 

If the scenarios discussed here have a non-infinitesimal 
chance of actually happening, then artificial intelligence 
and the opportunities and risks associated with it should 
be a global priority. The probability of a good outcome of 
Al research can be maximized through a number of mea- 
sures, including the following: If the scenarios discussed 


here have (a perhaps small, but) more than an infinites- 
imal chance of actually happening, then artificial intelli- 
gence and the opportunities and risks associated with it 
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should be a global priority. The probability of a good out- 
come of Al research can be maximised through the follow- 
ing measures, amongst others: 


Recommendation 5 — Information: An effective improvement in the safety of artificial intelligence research begins 
with awareness on the part of experts working on Al, investors, and decision-makers. Information on the risks asso- 
ciated with Al progress must, therefore, be made accessible and understandable to a wide audience. Organizations 
supporting these concerns include the Future of Humanity Institute (FHI) at the University of Oxford, the Machine In- 
telligence Research Institute (MIRI) in Berkeley, the Future of Life Institute (FLI) in Boston, as well as the Foundational 
Research Institute (FRI). = 


Recommendation 6 — Alsafety: Recent years have witnessed an impressive rise in investment into Al research [86], 
but research into Al safety has been comparatively slow. The only organization currently dedicated the theoretical 
and technical problems of Al safety as its top priority is the aforementioned MIRI. Grantors should encourage research 
projects to document the relevance of their work to Al safety, as well as the precautions taken within the research itself. 
At the same time, high-risk Al research should not be banned, as this would likely result in a rapid and extremely risky 
relocation of research to countries with lower safety standards. = 


Recommendation 7 — Global cooperation and coordination: Economic and military incentives create a competitive 
environment in which a dangerous Al arms race will almost certainly arise. In the process, the safety of Al research 
will be reduced in favor of more rapid progress and reduced cost. Stronger international cooperation can counter this 
dynamic. Ifinternational coordination succeeds, then a “race to bottom” in safety standards (through the relocation of 
scientific and industrial Al research) would also be avoided. 


Artificial consciousness 


Humans and many non-human animals have what is 
known as phenomenal consciousness—that is, they expe- 
rience themselves to be a human or a non-human animal 
with a subjective, first-person point of view [99]. They 
have sensory impressions, a (rudimentary or pronounced) 
sense of self, experiences of pain upon bodily damage, 
and the capacity to feel psychological suffering or joy (see 
for example the studies of depression in mice [100]). In 
short, they are sentient beings. Consequently, they can be 
harmed in a sense that is relevant to their own interests 
and perspective. In the context of Al, this leads to the fol- 
lowing question: Is it possible for the functional system of 
a machine to also experience a potentially painful “inner 
life”? The philosopher and cognitive scientist Thomas Met- 
zinger offers four criteria for the concept of suffering, all of 
which would apply to machines as well as animals: 


1. Consciousness. 
2. A phenomenal self-model. 


3. The ability to register negative value (thatis, violated 
subjective preferences) within the self-model. 


4. Transparency (that is, perceptions feel irrevocably 


“real”, thus forcing the system to self-identify with 
the content of its conscious self-model) [101, 102]. 


Two related questions have to be distinguished actually: 
firstly, whether machines could ever develop conscious- 
ness and the capacity for suffering at all; and secondly, if 
the answer to the first question is yes, which types of ma- 
chines (will) have consciousness. 

In addition to the above, two related questions have 
to be distinguished: Firstly, whether machines could tech- 
nically develop consciousness and the capacity for suffer- 
ing at all; Secondly, if the answer to the first question is 
yes, which types of machines (will) have consciousness. 
These two questions are being researched by philosophers 
and Al experts alike. A glance at the state of research re- 
veals that the first question is easier to answer than the 
second. There is currently substantial, but not total, con- 
sensus amongst experts that machines could in principle 
have consciousness, and that it is at least possible in neu- 
romorphic computers [103, 104, 105, 106, 107, 108, 109]. 
Such computers have hardware with the same functional 
organization as a biological brain [110]. The question of 
identifying which types of machines (besides neuromor- 
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phic computers) could have consciousness, however, is far 
more difficult to answer. The scientific consensus in this 
area is less clear [111]. For instance, it is disputed whether 
pure simulations (such as the simulated brain of the Blue 
Brain Project) could have consciousness. While some ex- 
perts are confident that this is the case [109, 105], others 
disagree [111, 112]. 

In view of this uncertainty among experts, it seems rea- 
sonable to take a cautious position: According to current 
knowledge, it is at least conceivable that many sufficiently 
complex computers, including non-neuromorphic ones, 
could be sentient. 

These considerations have far-reaching ethical conse- 
quences. If machines could have consciousness, then it 
would be ethically unconscionable to exploit them as a 
workforce, and to use them for risky jobs such as defusing 
mines or handling dangerous substances [4, p. 167]. If suf- 
ficiently complex Als will have consciousness and subjec- 
tive preferences with some probability, then similar ethi- 
cal and legal safety precautions to those used for humans 
and non-human animals will have to be met [113]. If, say, 
the virtual brain of the Blue Brain Project was to gain con- 
sciousness, then it would be highly ethically problematic 
to use it (and any potential copies or “clones”) for system- 
atic research of e.g. depression by placing it in depres- 


sive circumstances. Metzinger warns that conscious ma- 
chines could be misused for research purposes. Moreover, 
as “second class citizens”, they may lack legal rights and be 
exploited as dispensable experimental tools, all of which 
could be negatively reflected at the level of the machines’ 
inner experience [106]. This prospect is particularly worry- 
ing because it is conceivable that Als will be made in such 
huge numbers [4, 75] that in a worst-case scenario, there 
could be an astronomical number of victims, outnumber- 
ing any known catastrophe in the past. 

These dystopian scenarios point toward an important 
implication of technological progress: Even if we make 
only “minor” ethical mistakes (e.g. by erroneously clas- 
sifying certain computers as unconscious or morally in- 
significant), then by virtue of historically unprecedented 
technological power, this could result in equally unprece- 
dented catastrophes. If the total number of sentient be- 
ings rises drastically, we must ensure that our ethical val- 
ues and empirical estimates improve proportionally; a 
mere marginal improvement in either parameter will be 
insufficient to meet the greatly increased responsibility. 
Only by acknowledging the uncertain nature of possible 
machine consciousness can we begin to take appropri- 
ate cautionary measures in Al research, and thus hope to 
avoid any of the potential catastrophes described above. 


Recommendation 8 — Research: In order to make ethical decisions, it is important to have an understanding of which 
natural and artificial systems have the capacity for producing consciousness, and in particular for experiencing suffer- 
ing. Given the apparent level of uncertainty and disagreement within the field of machine consciousness, there is a 
pressing need to promote, fund, and coordinate relevant interdisciplinary research projects (comprising philosophy, 
neuroscience, and computer science). = 


Recommendation 9 — Regulation: It is already standard practice for ethics commissions to regulate experiments on 
living test subjects [114, 115]. In light of the possibility that neuromorphic computers and simulated beings could also 
develop consciousness, it is vital that research on these, too, is carried out under the strict supervision of ethics commis- 
sions. Furthermore, the (unexpected) creation of sentient artificial life should be avoided or delayed wherever possible, 
as the Als in question could—once created—be rapidly duplicated on a vast scale. In the absence of pre-existing legal 


representation and political interest in artificial sentience, this proliferation would likely continue unchecked. a 


Conclusion 


Already today, we are witnessing the spread of novel Al 
technologies with surprising potential. The Al technology 
currently behind driverless cars, Watson-assisted medical 
diagnosing, and US military drones will gradually become 
available for general use in the foreseeable future. It is 
crucial that carefully constructed legal frameworks are in 
place before this happens, so as to realize the potential of 
these technologies in ways that safely minimize any risks 
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of a negative overall development. 

The more progress is made in the field of Al technol- 
ogy, the more pressing a rational, far-sighted approach to 
the associated challenges becomes. Because political and 
legal progress tends to lag behind technological develop- 
ment, there is an especially large amount ofresponsibility 
resting on the individual researchers and developers who 
directly take part in any progress being made. 


Unfortunately, however, there are strong economic in- 
centives for the development of new technologies to take 
place as fast as possible without “wasting” time on expen- 
sive risk analyses. These unfavorable conditions increase 
the risk that we gradually lose our grip on the control of 
Al technology and its use. This should be prevented on 
all possible levels, including politics, the research itself, 
and in general by anyone whose work is relevant to the is- 
sue. A fundamental prerequisite to directing Al develop- 
ment along the most advantageous tracks possible will be 
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to broaden the field of Al safety. This way, it can be recog- 
nized not only among a few experts but in widespread pub- 
lic discourse as a great (perhaps the greatest) challenge of 
our age. 

As a final addition to the concrete recommendations 
given above, we would like to conclude by pleading that Al 
risks and opportunities be recognized as a global priority— 
akin to climate change, or the prevention of military 
conflicts—as soon as possible. 
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